Visual Relationship Detection With Visual-Linguistic Knowledge From Multimodal Representations

Visual relationship detection aims to reason over relationships among salient objects in images, which has drawn increasing attention the past few years. Inspired by human reasoning mechanisms, it is believed that external visual commonsense knowledge beneficial for of however rarely considered existing methods. In this paper, we propose a novel approach named Relational Visual-Linguistic Bidirectional Encoder Representations from Transformers (RVL-BERT), performs relational with both and language learned via self-supervised pre-training multimodal representations. RVL-BERT also uses an effective spatial module mask explicitly capture information objects. Moreover, our model decouples object recognition taking names directly, enabling be used on top any system. We show through quantitative qualitative experiments that, transferred modules, achieves competitive results two challenging datasets. The source code available at https://github.com/coldmanck/RVL-BERT.